Added new problem TD(0) policy evaluation update #517
Conversation
moe18
left a comment
moe18
left a comment
There was a problem hiding this comment.
had a small comment on the TD imp
questions/172_td0_value_function_update_for_single_episode/solution.py
Outdated
Show resolved
Hide resolved
|
@moe18 |
|
that makes sense so will push the TD Q |
moe18
left a comment
moe18
left a comment
There was a problem hiding this comment.
had a few small comments, sorry for the late response back
| @@ -0,0 +1,3 @@ | |||
| def td0_policy_evaluation(episode, V, pi, alpha): | |||
| for (s, a, r, s_next) in episode: | |||
| V[s] += alpha * (r + V[s_next] - V[s]) No newline at end of file | |||
There was a problem hiding this comment.
you need to return the V value
| }, | ||
| { | ||
| "test": "episode = [\n ('A', 'left', 5.0, 'B'),\n ('B', 'right', 0.0, 'C'),\n ('C', 'down', 1.0, 'terminal')\n]\nV = {'A': 0.0, 'B': 0.0, 'C': 0.0, 'terminal': 0.0}\npi = {'A': 'left', 'B': 'right', 'C': 'down'}\nalpha = 0.5\nV_updated = td0_policy_evaluation(episode, V, pi, alpha)\nprint({k: round(v, 2) for k, v in V_updated.items()})", | ||
| "expected_output": "{'A': 2.5, 'B': 0.5, 'C': 0.5, 'terminal': 0.0}" |
There was a problem hiding this comment.
the solution you provided gave this output
{'A': 2.5, 'B': 0.0, 'C': 0.5, 'terminal': 0.0}
| @@ -0,0 +1,23 @@ | |||
|
|
|||
| # Learn Section | |||
There was a problem hiding this comment.
no need to say learn section
moe18
left a comment
moe18
left a comment
There was a problem hiding this comment.
some small changes to the SARSA question, but looks good
| @@ -0,0 +1,15 @@ | |||
| { | |||
| "id": "173", | |||
| "title": "implement_the_SARSA_Algorithm_on_policy", | |||
There was a problem hiding this comment.
no need for '_' should just be Implement the SARSA Algorithm on policy
| }, | ||
| { | ||
| "test": "transitions = {\n ('A', 'x'): (0.0, 'terminal'),\n ('A', 'y'): (5.0, 'B'),\n ('B', 'z'): (2.0, 'terminal')\n}\ninitial_states = ['A']\nalpha = 0.4\ngamma = 0.9\nmax_steps = 3\nQ = sarsa_update(transitions, initial_states, alpha, gamma, max_steps)\nfor k in sorted(Q):\n print(f\"Q{str(k):15} = {Q[k]:.4f}\")", | ||
| "expected_output": "Q('A', 'x') = 0.0000\nQ('A', 'y') = 0.0000\nQ('B', 'z') = 0.0000" |
There was a problem hiding this comment.
got this output from the solution Q('A', 'x') = 0.0000 Q('A', 'y') = 0.0000
|
@moe18 |
2 participants
Added a new problem implementation for TD(0) Policy Evaluation. This function performs a single pass of value updates over an episode of
(state, action, reward, nextstate)transitions that follow a deterministic policyπ. Includes:1.Core implementation of TD(0) update rule.
2.Markdown version of algorithm.
3.Structured test cases with reasoning and expected outputs.